Hi,Currently, I am using the below-mentioned query to create a materialized view. It was working fine until yesterday in the DLT pipeline, but from today on, the below-provided code throws an error (com.databricks.sql.transaction.tahoe.ColumnMappingE...
Hello. Is there a way to enforce the length of a column in SQL? For example that a column has to be exactly 18 characters? Thank you!
we are facing similar issues while write into adls location delta format, after that we created on top delta location unity catalog tables. below format of data type length should be possible to change spark sql supported ?Azure SQL Spark ...
Need to understand this i got 2 sources which currently writes to the Single S3 bucket in csv and json format daily is it possible to ingest both the file types in a single build to ingest in Databricks, file size <300 mb ?or Alternately i should cre...
Rename and drop columns with Delta Lake column mapping. Hi all,Now databricks started supporting column rename and drop.Column mapping requires the following Delta protocols:Reader version 2 or above.Writer version 5 or above.Blog URL##Available in D...
Above mentioned feature is not working in the DLT pipeline. if the scrip has more than 4 columns
Hello all,We are building a data warehouse on Unity Catalog and we use the SHALLOW CLONE command to allow folks to spin up their own dev environments by light copying the prod tables. We also started using Liquid Clustering on our feature tables, tho...
Thanks Kaniz for your reply. I was able to get it make it work using runtime 14.0.Regards,
I have set up a Jupyter Notebook w/ PySpark connected to a Spark cluster, where the Spark instance is intended to perform writes to a Delta table.I'm observing that the Spark instance fails to complete the writes if the Jupyter Notebook doesn't have ...
I have raw call log data and the logs don't have a unique id number so I generate a uuid4 number when i load them using spark. Now I want to save the records to a table, and run a COPY INTO command every day to ingest new records. I am only appendi...
Is it possible to convert to or export as a .ipynb notebook?
You can rename somefile.dbc to somefile.zip and open it with the Windows File Explorer, however the .python files cannot be opened in vscode or pycharm
Hi,Greetings of the day, I am preparing for the Databricks Engineering Associate Certification and following the Databricks Academy V3 course. As part of this course, trying to create the DLT pipelines but couldn't able to see the DBAcademyDLT policy...
I had to go to Admin Settings > Advanced (Workspace settings) and enable "Cluster, Pool and Jobs Access Control". Afterwards I was able to create an instance pool.
I get this error when I run the Databricks training notebook%run ../Includes/Classroom-Setup-01 Resetting the learning environment: | dropping the catalog "***_53sh_da"...(0 seconds) Skipping install of existing datasets to "dbfs:/mnt/dbacademy-da...
Could you fixed it? I have the same problem
I have shape files with polygon/geometry info. I am exporting the file after Sedona integration with Kepler.I o/p file is in .html. I want to reduce the file size.Pls suggest in case any option is available.
Hi Team,I accidentally deleted our databricks workspace, which had all our artefacts and control plane, and was the primary resource for our team's working environment.Could anyone please help on priority, regarding the recovery/ restoration mechanis...
I am trying to copy a dashboard object from one workspace to another using API. I am using Get dashboard objects (/api/2.0/preview/sql/dashboards GET method), then Retrieve a definition (/api/2.0/preview/sql/dashboards/{dashboard_id} GET method) and ...
Hi @markusk, unfortunately the issue you're running into is a limitation of the outgoing SQL dashboard APIs. You can create blank dashboards, but you cannot programmatically insert existing definitions. The good news is we're currently running a prev...
Hello all!I've been working on integrating a Databricks Repos update API call to a DevOps Pipeline so that the Databricks local repo stays up to date with the remote staging branch (Pipeline executes whenever there's a new commit in to the staging br...
@BookerE1 I found it!. There was already another thread related to this problem and someone else helped me find the solution (Problem was the pool that I was using for the pipeline)This is the link to the other thread: https://community.databricks.co...
In List Pipelines API documentation, at response is specified that response is statuses Array of objects The list of events matching the request criteria. next_page_token string If present, a token to fetch the next page of events. But if we retri...
@costi9992 This has been documented now: https://docs.databricks.com/api/workspace/jobs/listruns
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
