- 1793 Views
- 3 replies
- 2 kudos
Resolved! How can we get AutoLoader to detect a file footer?
We are dealing with CSVs that have footers in them. When we have an empty file, the presence of this footer seems to impair the schema inferencing of AutoLoader, because of the footer.I know where is a header = true parameter, but I don't see a foote...
- 1793 Views
- 3 replies
- 2 kudos
- 2 kudos
To be clear, when you say footer are you referring to the last row of the tuple? e.g. Header = row 1, Footer = row_last.
- 2 kudos
- 1400 Views
- 2 replies
- 1 kudos
Resolved! Can we implement Unity Catalog table lifecycle?
I want to delete tables that haven't been selected or otherwise accessed for several months.I can see the Delta table history, but I can only catch the DDL or update/insert/delete and can't catch "select".I realized that the Unity Catalog insight, ht...
- 1400 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Renu_ ,I appreciate for your clear response. I now have a better understanding and will work with our admin team to develop a strategy.Thank you.
- 1 kudos
- 2674 Views
- 2 replies
- 0 kudos
Resolved! Concurency behavior with merge operations
Hi community,I have this case right now in project where i have to develop a solution that will prevent duplicate data from being ingested twice to delta lake. Some of our data suppliers at a rare occurence are sending us the same dataset in two diff...
- 2674 Views
- 2 replies
- 0 kudos
- 0 kudos
Your idea of using a log table to track processed ingestions and leveraging a MERGE operation in your pipeline is a sound approach for preventing duplicate data ingestion into Delta Lake. Delta Lake's ACID transactions and its support for concurrency...
- 0 kudos
- 3380 Views
- 2 replies
- 0 kudos
DBFS Permissions
if there is permission control on the folder/file level in DBFS.e.g. if a team member uploads a file to /Filestore/Tables/TestData/testfile, could we mask permissions on TestData and/or testfile?
- 3380 Views
- 2 replies
- 0 kudos
- 830 Views
- 1 replies
- 0 kudos
NOT ABLE TO ATTACH CLUSTRE
notebook detached-exception when creating execution context:java.until.concurrent.timeoutexceoption:timed out after 15 seconds
- 830 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @sahil3 Try detach and re-attaching the notebook to the notebook. Please note that this will clear the state of the notebook.if the issue persists, try restarting the cluster.Best,
- 0 kudos
- 2554 Views
- 3 replies
- 1 kudos
Resolved! How to use read_kafka() SQL with secret()?
Hi,I want to read data from the Azure Event Hub using SQL.Can someone please give me an executable example where you can also use the connection string from the event hub using the SQL function secret(), for example?This is what i tried but it Databr...
- 2554 Views
- 3 replies
- 1 kudos
- 1 kudos
I found the solution und could successfully establish a connection to Event-Hub. SELECT cast(value as STRING) as raw_json, current_timestamp() as processing_time FROM read_kafka( bootstrapServers => '<YOUR EVENT-HUB NAMESPACE>.servicebus.windows.n...
- 1 kudos
- 5644 Views
- 5 replies
- 0 kudos
On-behalf-of token creation for service principals is not enabled for this workspace
Hi AllI just wanted to create PAT for Databricks Service Principle but getting below code while hitting API or using CLI - Please help me to create PAT for the same.#dataengineering #databricks
- 5644 Views
- 5 replies
- 0 kudos
- 0 kudos
You can generate the token while logged in as the Service Principle via the Azure CLI in a Command Prompt window. To do so, make sure to install the Azure CLI and the Databricks CLI with it.Install the Azure CLI for Windows | Microsoft LearnInstall ...
- 0 kudos
- 2659 Views
- 1 replies
- 0 kudos
Reading CloudWatch Logs from AWS Kinesis
If you have AWS CloudWatch subscribed to write out logs to AWS Kinesis, the Kinesis stream is base64 encoded and the CloudWatch logs are GZIP compressed. The challenge we faced was how to address that in pyspark to be able to read the data. We were ...
- 2659 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you so much, this is very helpful
- 0 kudos
- 2794 Views
- 3 replies
- 3 kudos
Resolved! What is the difference between spark inferschema and cloudFiles.inferColumnTypes?
We have been using spark.read with inferSchema = True to validate AutoLoader schema inferencing. But I have a suspicion that they do these differently from each other and may not always yield the identical results.Has anyone ever answered this questi...
- 2794 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @BF7 Yes — there is a difference between how spark.read(...).option("inferSchema", "true")and Auto Loader's schema inference (cloudFiles.schemaHints, cloudFiles.inferColumnTypes, etc.) work.They are not guaranteed to produce identical results,Key ...
- 3 kudos
- 1969 Views
- 3 replies
- 1 kudos
Resolved! springml sftp with spark 3.x
Is there a version of springml spark-sftp that works with spark 3.x and scala 2.12? If so can you point me to it or how to load it in my compute?
- 1969 Views
- 3 replies
- 1 kudos
- 1 kudos
For Python you might want to look at Paramiko, it seems that it might be an option. You could also look at ETL tools like Airbyte, Rivery, CData, etc.
- 1 kudos
- 4276 Views
- 3 replies
- 0 kudos
SQLSTATE: XX000 The Spark SQL phase planning failed with an internal error.
Hello everyone,I am currently working with a SQL Warehouse and have been getting the following error message:[INTERNAL_ERROR ] The Spark SQL phase planning failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, re...
- 4276 Views
- 3 replies
- 0 kudos
- 0 kudos
I have tried to simplify the query as much as possible to see if that helps but the bug still persists. The problem should be something with the way Databricks treats columns passed as arguments for a function.I tried these queries:select * FROM VALU...
- 0 kudos
- 11185 Views
- 15 replies
- 3 kudos
API for Restarting Individual Failed Tasks within a Job?
Hi everyone,I'm exploring ways to streamline my workflow in Databricks and could really use some expert advice. In my current setup, I have a job (named job_silver) with multiple tasks (e.g., task 1, task 2, task 3). When one of these tasks fails—say...
- 11185 Views
- 15 replies
- 3 kudos
- 3 kudos
Hey @minhhung0507 - quick question - what is the cluster type you're using to run your workflow?I'm using a shared, interactive cluster, so I'm passing the parameter {'existing_cluster_id' : task['existing_cluster_id']}in the payload. This parameter ...
- 3 kudos
- 2304 Views
- 4 replies
- 0 kudos
Resolved! Debugging jobs/run-now endpoint
I am not being able to run jobs/runnow endpoint. I am getting an error asError fetching files: 403 - {"error_code":"PERMISSION_DENIED","message":"User xxxx-dxxxx-xxx-xxxx does not have Manage Run or Owner or Admin permissions on job 437174060919465",...
- 2304 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @smpa01 - The PAT you're using belongs to Service Principal or your personal token? If SP, it should have permissions to run the DBX workflow. Let me know if any questions.
- 0 kudos
- 3183 Views
- 5 replies
- 5 kudos
Resolved! Dlt pipeline showing legacy , even though all things are latest
Some of old dlt pipelines in my databricks workspace are showing legacy,I am using serverless pipeline with mode - preview.Anything which I missed ?
- 3183 Views
- 5 replies
- 5 kudos
- 1065 Views
- 1 replies
- 0 kudos
Writing files using multithreading to dbfs
Hello,I am reading in xml files from AWS S3 and storing them on dbfs:/ using multithreaded code. The code itself seems to be fine as for the first +- 100 000 files it works without issues and I can see the data arriving on DBFS.However it will always...
- 1065 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @daan_dw I think this issue mainly comes from using multithreading to handle XML files while interacting with both S3 and DBFS. When the thread count gets too high, it likely causes race conditions.To avoid this:Try reducing the number of threads....
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 573 | |
| 349 | |
| 287 |
