Trino Select from public aws S3 storage anonymously (hive connector)
My hive connector config, hive_aws.properties:
connector.name=hive
hive.metastore.uri=thrift://hive-metastore:9083
fs.native-s3.enabled=true
s3.endpoint=https://s3.amazonaws.com
s3.region=us-east-1
(I understand I simply have to provide no credentials for anonymous S3 access?)
Then trying to create a Table:
CREATE TABLE hive_aws.default.noaa_ghcn_2024 (
id VARCHAR,
date varchar,
data_value INT,
m_flag VARCHAR,
q_flag VARCHAR,
s_flag VARCHAR,
obs_time VARCHAR,
year int,
element VARCHAR
)
WITH (
format = 'PARQUET',
external_location = 's3://noaa-ghcn-pds/parquet/by_year/', -- I have also tried s3://noaa-ghcn-pds.s3.amazonaws.com/parquet/by_year/ and not providing s3.endpoint=https://s3.amazonaws.com hive_aws.properties
partitioned_by = ARRAY['YEAR', 'ELEMENT']
)
Whatever I try, I get this error, note the root cause -> "Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain":
io.trino.spi.TrinoException: External location is not a valid file system URI: s3://noaa-ghcn-pds.s3.amazonaws.com/parquet/by_year/
at io.trino.plugin.hive.HiveMetadata.checkExternalPathAndCreateIfNotExists(HiveMetadata.java:1429)
at io.trino.plugin.hive.HiveMetadata.createTable(HiveMetadata.java:1109)
at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.createTable(ClassLoaderSafeConnectorMetadata.java:461)
at io.trino.tracing.TracingConnectorMetadata.createTable(TracingConnectorMetadata.java:388)
at io.trino.metadata.MetadataManager.createTable(MetadataManager.java:878)
at io.trino.tracing.TracingMetadata.createTable(TracingMetadata.java:436)
at io.trino.execution.CreateTableTask.internalExecute(CreateTableTask.java:306)
at io.trino.execution.CreateTableTask.execute(CreateTableTask.java:128)
at io.trino.execution.CreateTableTask.execute(CreateTableTask.java:94)
at io.trino.execution.DataDefinitionExecution.start(DataDefinitionExecution.java:152)
at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:284)
at io.trino.dispatcher.LocalDispatchQuery.startExecution(LocalDispatchQuery.java:150)
at io.trino.dispatcher.LocalDispatchQuery.lambda$waitForMinimumWorkers$1(LocalDispatchQuery.java:134)
at io.airlift.concurrent.MoreFutures.lambda$addSuccessCallback$0(MoreFutures.java:570)
at io.airlift.concurrent.MoreFutures$3.onSuccess(MoreFutures.java:545)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
at io.trino.$gen.Trino_478____20251201_172151_2.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: io.trino.filesystem.TrinoFileSystemException: Failed to list location: s3://noaa-ghcn-pds.s3.amazonaws.com/parquet/by_year/
at io.trino.filesystem.s3.S3FileSystem.listObjects(S3FileSystem.java:270)
at io.trino.filesystem.s3.S3FileSystem.listFiles(S3FileSystem.java:240)
at io.trino.filesystem.s3.S3FileSystem.directoryExists(S3FileSystem.java:279)
at io.trino.filesystem.switching.SwitchingFileSystem.directoryExists(SwitchingFileSystem.java:123)
at io.trino.filesystem.tracing.TracingFileSystem.lambda$directoryExists$0(TracingFileSystem.java:129)
at io.trino.filesystem.tracing.Tracing.withTracing(Tracing.java:51)
at io.trino.filesystem.tracing.TracingFileSystem.directoryExists(TracingFileSystem.java:129)
at io.trino.plugin.hive.HiveMetadata.checkExternalPathAndCreateIfNotExists(HiveMetadata.java:1419)
... 19 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130)
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:134)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.resolveIdentity(AwsCredentialsProvider.java:54)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.lambda$trySelectAuthScheme$6(S3AuthSchemeInterceptor.java:169)
at software.amazon.awssdk.core.internal.util.MetricUtils.reportDuration(MetricUtils.java:81)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.trySelectAuthScheme(S3AuthSchemeInterceptor.java:169)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.selectAuthScheme(S3AuthSchemeInterceptor.java:87)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.beforeExecution(S3AuthSchemeInterceptor.java:67)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.lambda$beforeExecution$1(ExecutionInterceptorChain.java:59)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.beforeExecution(ExecutionInterceptorChain.java:59)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.runInitialInterceptors(AwsExecutionContextBuilder.java:319)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:155)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:67)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:8724)
at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:154)
at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:145)
at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)
at software.amazon.awssdk.core.pagination.sync.PaginatedItemsIterable$ItemsIterator.(PaginatedItemsIterable.java:58)
at software.amazon.awssdk.core.pagination.sync.PaginatedItemsIterable.iterator(PaginatedItemsIterable.java:48)
at java.base/java.lang.Iterable.spliterator(Iterable.java:101)
at software.amazon.awssdk.core.pagination.sync.SdkIterable.stream(SdkIterable.java:34)
at io.trino.filesystem.s3.S3FileSystem.listObjects(S3FileSystem.java:263)
... 26 more
I also tried by setting:
s3.signer-type=Aws4UnsignedPayloadSigner
... which did not help.
When I read the error message seems to indicate that providers are looked at for the AwsCredentialsProviderChain in this order:
- SystemPropertyCredentialsProvider
- EnvironmentVariableCredentialsProvider
- WebIdentityTokenCredentialsProvider
- ProfileCredentialsProvider
- ContainerCredentialsProvider
- InstanceProfileCredentialsProvider
Am I correct to assume that as the AnonymousCredentialsProvider is nowhere in this chain anonymous access to S3 Storage is not possible anymore with Trino?
If that is the case: Could it be added to the end of that chain?
I also tried:
s3.credentials-provider=software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider
Looking at TrinoS3FileSystem.java I would have thought this could work but when starting Trino I get:
2025-12-02T14:26:47.990Z ERROR main io.trino.server.Server Configuration errors:
1) Error: Configuration property 's3.credentials-provider' was not used