- 4648 Views
- 2 replies
- 1 kudos
Best practices for Liquid clustering and z-ordering for existing streaming delta tables
Hello, I have been tasked to optimize some of our existing tables in delta lake in Databricks and I was able to perform following clause on some of our delta tables but I wasn't able to execute the same clause against some of our streaming tables. AL...
- 4648 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @filipniziol , thank you for your quick response. Not executing the alter statement against the DLT table directly and instead adjusting the pipeline configuration approach makes sense. Regarding the liquid clustering on the DLT tables, I assume i...
- 1 kudos
- 2939 Views
- 5 replies
- 0 kudos
Code Optimization
1. I have alot of data transformation which result in df1, then df1 is starting point for 10 different transformations path. At first I tried to use .cache() and .count() on df1 but it was very slow. I changed caching to saving df1 as delta table and...
- 2939 Views
- 5 replies
- 0 kudos
- 0 kudos
It is hard to tell why the transformations are slow without any details.You would need to share the file you are trying process and the code you are running.
- 0 kudos
- 6114 Views
- 4 replies
- 2 kudos
Resolved! Delta Live Tables (Triggered Mode), how and in what order is the data processed?
Hey!I have set up a delta live tables pipeline with bronze and silver tables.I have one bronze tables which ingest data from a storage account using autoloader.Multiple files are uploaded at once in the storage account.My silver tables read and proce...
- 6114 Views
- 4 replies
- 2 kudos
- 2 kudos
Thanks @szymon_dybczak for your reply!Do you maybe have a link to documentation for this? I didn't find a lot of info about this and would love to read more on how Delta Live Tables work in Triggered vs Streaming mode
- 2 kudos
- 1728 Views
- 1 replies
- 0 kudos
Databricks Certified Data Engineer Professional
Hello Team,I have experience in the databricks.But most of the work i had on analytics side.So i need to get certified Databricks Certified Data Engineer Professional.So please any one can help me to get the content to read and prepare for the exams....
- 1728 Views
- 1 replies
- 0 kudos
- 0 kudos
@RohitKulkarni here is the link with all the details https://www.databricks.com/learn/certification/data-engineer-professional
- 0 kudos
- 9688 Views
- 3 replies
- 1 kudos
ANTLR Tool version 4.8 used for code generation does not match the current runtime version 4.9.3
I have a Databricks job that runs daily at 14:00 IST and typically finishes in about 2 hours. However, yesterday, the job got stuck and continued running indefinitely. After exceeding 5 hours, I canceled it and reran the job, which then completed suc...
- 9688 Views
- 3 replies
- 1 kudos
- 1 kudos
When you navigate to the corresponding cluster, you'll see, "Event log", "Spark UI" and "Driver logs". There, you should find all the needed information.
- 1 kudos
- 1722 Views
- 1 replies
- 0 kudos
Resolved! Compute cluster not working after installing ojdbc
Hi ,I have a Databricks 12.2 LTS compute cluster which as expected.I need to establish conn to Oracle, once ojdbc11 Driver jar is installed. Cluster does not execute any cell (even print statement) and stuck in waiting state.In Driver logs it shows:'...
- 1722 Views
- 1 replies
- 0 kudos
- 0 kudos
Issue is resolved.Installing drivers directly collide with Databricks internal runtime libraries.We have used init script to copy jar from workspace to databricks/jars/.
- 0 kudos
- 1796 Views
- 1 replies
- 0 kudos
Best Approach to Store Data in Azure Gov Cloud Workspace without Unity Catalog
Our team is using a workspace Azure Gov Cloud. We would like to download files from an external source into our Workspace. Since Unity Catalog is not enabled in Azure Gov Cloud, we’re looking for the best approach for securely storing data in our wor...
- 1796 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello!For securely storing sensitive data in your Azure Gov Cloud workspace without Unity Catalog, use Azure Blob Storage with encryption, Azure Data Lake Storage with AAD access control, or Azure Key Vault for secrets. Managed identities can also he...
- 0 kudos
- 10475 Views
- 6 replies
- 5 kudos
SAP Business Object(BO) Integration with Databricks
Hi Team,We are doing an analysis on SAP Business object to connect with databricks and built a report on top of the data in the data lakehouse. In our current architecture we have delta tables on top of S3 storage. Please let us know any connectors/d...
- 10475 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @HB83 , Can I know which version of BO are you using? We have a similar requirement.
- 5 kudos
- 10945 Views
- 4 replies
- 2 kudos
Resolved! How to use autoloader with csv containing spaces in attribute names?
I am attempting to use autoloader to add a number of csv files to a delta table. The underlying csv files have spaces in the attribute names though (i.e. 'Account Number' instead of 'AccountNumber'). When I run my autoload, I get the following error ...
- 10945 Views
- 4 replies
- 2 kudos
- 2 kudos
@Hubert Dudek​ thanks for your response! I was able to use what you proposed above to generate the schema. The issue is that the schema sets all attributes to STRING values and renames them numerically ('_c0', '_c1', etc.). Although this allows us to...
- 2 kudos
- 22566 Views
- 12 replies
- 7 kudos
dataframe takes unusually long time to save as a delta table using sql for a very small dataset with 30k rows. It takes around 2hrs. Is there a solution for this problem?
I am trying to save a dataframe after a series of data manipulations using Udf functions to a delta table. I tried using this code( df .write .format('delta') .mode('overwrite') .option('overwriteSchema', 'true') .saveAsTable('output_table'))but this...
- 22566 Views
- 12 replies
- 7 kudos
- 7 kudos
You should also look into the sql plan if the writing phase is indeed the part that is taking time. Since spark works on lazy evaluation, there might be some other phase that might be taking time
- 7 kudos
- 4496 Views
- 1 replies
- 0 kudos
SQL function refactoring into Databricks environment
Hello all,I'm currently working on importing some SQL functions from Informix Database into Databricks using Asset Bundle deploying Delta Live Table to Unity Catalog. I'm struggling importing a recursive one, there is the code :CREATE FUNCTION "info...
- 4496 Views
- 1 replies
- 0 kudos
- 16002 Views
- 7 replies
- 4 kudos
kafkashaded.org.apache.kafka.common.errors.TimeoutException: topic-downstream-data-nonprod not present in metadata after 60000 ms.
I am facing an error when trying to write data on Kafka using spark stream.#Extract source_stream_df= (spark.readStream .format("cosmos.oltp.changeFeed") .option("spark.cosmos.container", PARM_CONTAINER_NAME) .option("spark.cosmos.read.inferSchema.en...
- 16002 Views
- 7 replies
- 4 kudos
- 4 kudos
What event hub namespace you were using?I had same problem and resolved by changing pricing plan from basic to standard as Kafka apps is not supporting in basic planLet me know if you had anything else. Thanks
- 4 kudos
- 13965 Views
- 6 replies
- 0 kudos
Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport=ON
I have added a connection to a MySql database in Azure, and I have created a foreign catalog in Databricks. But when I go to query the database I get the following error;Connections using insecure transport are prohibited while --require_secure_trans...
- 13965 Views
- 6 replies
- 0 kudos
- 0 kudos
I was unable to use serverless compute with Azure MySQL. I did have a meeting with tech support at databricks and the conclusion at the time was - if your Azure MySql is not publically accessible, you cannot connect to it from serverless compute.I'm ...
- 0 kudos
- 3941 Views
- 3 replies
- 1 kudos
Resolved! Error when query event log: Cannot read more than one event logs in the same query
I try to follow the instructions of Monitor Delta Live Tables pipelines to query dlt.expection log.Here is the simple code version I copied from the Querying the event log section:CREATE TEMPORARY LIVE VIEW event_log_raw AS (SELECT * FROM event_log(T...
- 3941 Views
- 3 replies
- 1 kudos
- 1 kudos
H @szymon_dybczak , thank you for the advice. After investigating the problem has been solved. The error message mentioned in the title is not the key, the below error message “A pipeline with a different id is already registered in this Credential S...
- 1 kudos
- 4624 Views
- 3 replies
- 0 kudos
DAB run
Hello All,I am running this command : databricks bundle run -t dev dltPpl_job --debugBundle name: dltPpl. Bundle was generated using: databricks bundle init --target devError message: Error: exit status 1Failed to marshal state to json: unsupported a...
- 4624 Views
- 3 replies
- 0 kudos
- 0 kudos
@Retired_mod I was trying 'pip install --upgrade databricks' before but after upgrading the version with 'brew upgrade databricks it worked. Thanks for the help!
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
