- 1709 Views
- 3 replies
- 3 kudos
We are using the Databricks managed MLflow to log experiment runs for quite some time already and never experienced issues. However, now we seem to have encountered a bug in the associated Databricks UI.
We observe the following behavior when we keep adding new runs to an experiment:- In the beginning, the runs are still displayed correctly in the UI.- After a certain number of total runs, the following bug occurs in the UI: - In the UI, there are ...
- 1709 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Timo Burmeister​ Apologies for the delay! I went through the video, does it happen all the time? I see after sorting it with different filter the list appears.
- 3 kudos
- 7791 Views
- 3 replies
- 0 kudos
Error loading MANAGED table in unity catalog delta lake on azure. Anyone seen this issue? "ErrorClass=INVALID_PARAMETER_VALUE] Input path <file system name>.dfs.core.windows.net overlaps with other external tables"
00007160: 2023-01-30T14:22:06 [TARGET_LOAD ]E: Failed (retcode -1) to execute statement: 'COPY INTO `e2underwriting_dbo`.`product` FROM(SELECT cast(_c0 as INT) as `ProductID`, _c1 as `ShortName`, cast(_c2 as INT) as `Status`, cast(_c3 as TIMESTA...
- 7791 Views
- 3 replies
- 0 kudos
- 0 kudos
we have solved this issue related to Qlik replicate copying data into delta table
- 0 kudos
- 1371 Views
- 1 replies
- 1 kudos
- 1371 Views
- 1 replies
- 1 kudos
- 1 kudos
You can ensure there is always an active run of your Databricks job with the new continuous trigger type. https://docs.databricks.com/workflows/jobs/jobs.html#continuous-jobs
- 1 kudos
- 11364 Views
- 9 replies
- 3 kudos
Resolved! Can't write / overwrite delta table with error: oxxxx.saveAsTable. (Driver Error: OutOfMemory)
Current Cluster Config:Standard_DS3_v2 (14GB, 4 Cores) 2-6 workersStandard_DS3_v2 (14GB, 4Cores) for driverRuntime: 10.4x-scala2.12We want to overwrite a temporary delta table with new records. The records will be load by another delta table and tran...
- 11364 Views
- 9 replies
- 3 kudos
- 3 kudos
Hi,thank you for your help!We tested the configuration settings and it runs without any errors.Could you give us some more information, where we can find some documentation about such settings. We searched hours to fix our problem. So we contacted th...
- 3 kudos
- 5195 Views
- 2 replies
- 2 kudos
Resolved! How limit input rate reading delta table as stream?
Hello to everyone!I am trying to read delta table as a streaming source using spark. But my microbatches are disbalanced - one very small and the other are very huge. How I can limit this? I used different configurations with maxBytesPerTrigger and m...
- 5195 Views
- 2 replies
- 2 kudos
- 2 kudos
besides the parameters you mention, I don't know of any other which controls the batch size.did you check if the delta table is not horribly skewed?
- 2 kudos
- 3637 Views
- 5 replies
- 2 kudos
Are there any plans to add functions on the partition by fields of a delta table definition such as day() ? A similar capability exists in iceberg.
Benefit: This will help simplify the where clauses of the consumers of the tables? Just query on the main date field if I need all the data for a day. Not an extra day field we had to make.
- 3637 Views
- 5 replies
- 2 kudos
- 2 kudos
@Ryan Hager​ , yes it is possible using AUTO GENERATED COLUMNS since delta lake 1.2For example, you can automatically generate a date column (for partitioning the table by date) from the timestamp column; any writes into the table need only specify t...
- 2 kudos
- 3559 Views
- 4 replies
- 3 kudos
Implementation of Late arriving dimension in databricks
Hi Team, Can you please suggest to me how to implement the late arriving dimension or early arriving fact with examples or any sample script for reference? I have to implement the same using pyspark.Thanks.
- 3559 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Hare Krishnan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 3 kudos
- 3566 Views
- 3 replies
- 2 kudos
Resolved! Passed the Fundamentals of the Databricks Lakehouse Platform Accreditation, but no badge recieved. Tried "https://v2.accounts.accredible.com/retrieve-credentials?" showing no badge.
Passed the Fundamentals of the Databricks Lakehouse Platform Accreditation, but no badge recieved. Tried "https://v2.accounts.accredible.com/retrieve-credentials?" showing no badge.
- 3566 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Ranjeet Ahlawat​ ,Congratulations on the certification. For any certification you take in the databricks you will be receiving the certificate and the badge in 24-48 hours and sometimes in lesser time as well. All the best for your future certifi...
- 2 kudos
- 4630 Views
- 7 replies
- 0 kudos
Cannot reset password, no support
I cannot log in to my Databricks community account. I have already tried to receive support and no real support has been given. I attempt to reset my password, the link gets sent, but once I enter the new password it gets stuck permanently loading. I...
- 4630 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @Ahmet Korkmaz​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 0 kudos
- 21562 Views
- 14 replies
- 15 kudos
Resolved! How to convert records in Azure Databricks delta table to a nested JSON structure?
Let's say I have a delta table in Azure databricks that stores the staff details (denormalized). I wanted to export the data in the JSON format and save it as a single file on a storage location. I need help with the databricks sql query to group/co...
- 21562 Views
- 14 replies
- 15 kudos
- 1904 Views
- 3 replies
- 3 kudos
Where is the learning material to get Fundamentals of the Databricks Lakehouse Platform Accreditation?
Please provide me some information about how to get the martial to pass Fundamentals of the Databricks Lakehouse Platform Accreditation?
- 1904 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Shanthala Baleer​,Just a friendly follow-up. Are you still looking for help? adding @Vidula Khanna​ for visibility
- 3 kudos
- 1418 Views
- 1 replies
- 0 kudos
How to restart snowflake connector?
After using spark.read.format("snowflake").options(**options).option("dbtable", "table_name").load() to read a table from Snowflake, when I then change the table from Snowflake and read it again, it gives me the first version of the table. I have wor...
- 1418 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, that would work. However, it is a longish Snowflake query producing a number of tables that are all called by the Databricks notebook, so it requires quite a few changes. I'll use this alternative if I automate the process. However, I think this...
- 0 kudos
- 6617 Views
- 5 replies
- 7 kudos
Resolved! Why was SQL formatting removed inside spark.sql functions? Now it looks like a plain string.
Previously we were able to see SQL queries inside spark.sql() like this:But now it just looks like a plain string: I know it's not a big issue, but it's still annoying to have to code in SQL while having it all be blue, it makes debugging more cumber...
- 6617 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @Emilio Garza​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 7 kudos
- 4799 Views
- 4 replies
- 0 kudos
Creating a spot only single-node job compute cluster policy
Hi there,I need some help creating a new cluster policy that utilizes a single spot-instnace server to complete a job. I want to set this up as a job-compute to reduce costs and also utilize 1 spot instance.The jobs I need to ETL are very short and c...
- 4799 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Avkash Kana​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 0 kudos
- 2167 Views
- 4 replies
- 0 kudos
how to optimize the runtime in 10.4 cluster
i am loading the 1billion data from spark dataframe into target table, but in the 7.3 cluster it takes 3 hours to complete but after migrated to 10.4 cluster its taking 8 hours to complete , how can i reduce the time duration​
- 2167 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Mohammed sadamusean​,Could you provide more details on what are you doing? What type of transformations/actions are you doing? whats your source and sink? batch or streaming? all that information will help.
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
