Hi All Facing an issue while creating the delta log on minio

Question

Hi All Facing an issue while creating the delta log on minio server from Java standalone jar Works fine with my local MinIO server running on docker container but not when I deploy it to my staging env ```org apache hadoop fs s3a AWSClientIOException getFileStatus on <s3a nifi test delta log last checkpoint> com amazonaws SdkClientException Unable to execute HTTP request nifi test minio commonservices svc cluster local Name or service not known Unable to execute HTTP request nifi test minio commonservices svc cluster local Name or service not known at org apache hadoop fs s3a S3AUtils translateException S3AUtils java 214 ~ hadoop aws 3 3 4 jar na at org apache hadoop fs s3a S3AUtils translateException S3AUtils java 175 ~ hadoop aws 3 3 4 jar na at org apache hadoop fs s3a S3AFileSystem s3GetFileStatus S3AFileSystem java 3799 ~ hadoop aws 3 3 4 jar na at org apache hadoop fs s3a S3AFileSystem innerGetFileStatus S3AFileSystem java 3688 ~ hadoop aws 3 3 4 jar na at org apache hadoop fs s3a S3AFileSystem extractOrFetchSimpleFileStatus S3AFileSystem java 5401 ~ hadoop aws 3 3 4 jar na at org apache hadoop fs s3a S3AFileSystem open S3AFileSystem java 1465 ~ hadoop aws 3 3 4 jar na at org apache hadoop fs s3a S3AFileSystem open S3AFileSystem java 1441 ~ hadoop aws 3 3 4 jar na at org apache hadoop fs FileSystem open FileSystem java 976 ~ hadoop common 3 3 4 jar na at io delta storage HadoopFileSystemLogStore read HadoopFileSystemLogStore java 46 ~ delta storage 2 2 0 jar 2 2 0 at io delta standalone internal storage DelegatingLogStore read DelegatingLogStore scala 83 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal Checkpoints loadMetadataFromFile Checkpoints scala 136 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal Checkpoints loadMetadataFromFile Checkpoints scala 147 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal Checkpoints loadMetadataFromFile Checkpoints scala 147 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal Checkpoints loadMetadataFromFile Checkpoints scala 147 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal Checkpoints lastCheckpoint Checkpoints scala 110 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal Checkpoints lastCheckpoint$ Checkpoints scala 109 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal DeltaLogImpl lastCheckpoint DeltaLogImpl scala 42 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal SnapshotManagement getSnapshotAtInit SnapshotManagement scala 218 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal SnapshotManagement $init$ SnapshotManagement scala 37 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal DeltaLogImpl lt init gt DeltaLogImpl scala 47 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal DeltaLogImpl$ apply DeltaLogImpl scala 263 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal DeltaLogImpl$ forTable DeltaLogImpl scala 241 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone internal DeltaLogImpl forTable DeltaLogImpl scala ~ delta standalone 2 13 0 6 0 jar 0 6 0 at io delta standalone DeltaLog forTable DeltaLog java 164 ~ delta standalone 2 13 0 6 0 jar 0 6 0 at com siai delta delta lake DeltaLogRepository lt init gt DeltaLogRepository java 39 ~ classes na at com siai delta delta lake MinioDeltaLakeRepository writeToDeltaLake MinioDeltaLakeRepository java 178 ~ classes na at com siai delta delta lake DeltaLakeController createDeltaTable DeltaLakeController java 55 ~ classes na at jdk internal reflect GeneratedMethodAccessor59 invoke Unknown Source ~ na na at java base jdk internal reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43 ~ na na at java base java lang reflect Method invoke Method java 564 ~ na na at org springframework web method support InvocableHandlerMethod doInvoke InvocableHandlerMethod java 205 ~ spring web 5 3 12 jar 5 3 12 at org springframework web method support InvocableHandlerMethod invokeForRequest InvocableHandlerMethod java 150 ~ spring web 5 3 12 jar 5 3 12 at org springframework web servlet mvc method annotation ServletInvocableHandlerMethod invokeAndHandle ServletInvocableHandlerMethod java 117 ~ spring webmvc 5 3 12 jar 5 3 12 at org springframework web servlet mvc method annotation RequestMappingHandlerAdapter invokeHandlerMethod RequestMappingHandlerAdapter java 895 ~ spring webmvc 5 3 12 jar 5 3 12 at org springframework web servlet mvc method annotation RequestMappingHandlerAdapter handleInternal RequestMappingHandlerAdapter java 808 ~ spring webmvc 5 3 12 jar 5 3 12 at org springframework web servlet mvc method AbstractHandlerMethodAdapter handle AbstractHandlerMethodAdapter java 87 ~ spring webmvc 5 3 12 jar 5 3 12 at org springframework web servlet DispatcherServlet doDispatch DispatcherServlet java 1067 ~ spring webmvc 5 3 12 jar 5 3 12 at org springframework web servlet DispatcherServlet doService DispatcherServlet java 963 ~ spring webmvc 5 3 12 jar 5 3 12 at org springframework web servlet FrameworkServlet processRequest FrameworkServlet java 1006 ~ spring webmvc 5 3 12 jar 5 3 12 at org springframework web servlet FrameworkServlet doPost FrameworkServlet java 909 ~ spring webmvc 5 3 12 jar 5 3 12 ```

Omkar · Answer

As per the error you ve shared it looks like to be an issue with the DNS resolution of your service ```Unable to execute HTTP request nifi test minio commonservices svc cluster local Name or service not known``` You may want to check if the service URL `nifi test minio commonservices svc cluster local` is valid and accessible by the minio server in your staging env

Sireesha Madabhushi · Answer

No actually it was a hadoop conf setting that I had to change and switch to hadoop aws 3 2 4 This will disable the way of getting access through virtual hosting of individual buckets in MinIO ```conf set fs s3a path style access true ```

Omkar · Answer

Indeed S3 path style access removes the dependency on setting up DNS Glad to know it s resolved +1 skin tone 3