- 10368 Views
- 4 replies
- 3 kudos
Load CSV files with slightly different schemas
I have a set of CSV files generated by a system, where the schema has evolved over the years. Some columns have been added, and at least one column has been renamed in newer files. Is there any way to elegantly load these files into a dataframe? I ha...
- 10368 Views
- 4 replies
- 3 kudos
- 3 kudos
For reference - for anybody struggling with the same issues. All online examples using auto loader are written as one block statement on the form: (spark.readStream.format("cloudFiles") .option("cloudFiles.format", "csv") # The schema location di...
- 3 kudos
- 2657 Views
- 2 replies
- 2 kudos
Perform database operations like INSERT, and MERGE using the backend service (golang)
I would like to have an example or reference documentation on how we can create a live (capable of real-time data persistence) pipeline from an in-house golang (backend) service to the databricks tables.I would give a bit more detail on what the gola...
- 2657 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Harikrishnan G​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 2 kudos
- 8723 Views
- 2 replies
- 3 kudos
Unable to read data from delta table using a python script after the table properties change.
Renamed the Column to include () in a delta table and set the table properties to the below ​ALTER TABLE test_table SET TBLPROPERTIES ( 'delta.minReaderVersion' = '2', 'delta.minWriterVersion' = '5', 'delta.columnMapping.mode' = 'name' ) However w...
- 8723 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Lakshmi Jayaraman​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 3 kudos
- 1531 Views
- 2 replies
- 1 kudos
unable to login Databricks partner account same id and password as Acadmey Account.
Unable to login Databric partner academy account. same as Acadmey give me a link to verify my mail to databric partner Account. I have try soo many time but unable to verify and login to databric partner Account .
- 1531 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Firoz Ahmad Ansari​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 1 kudos
- 7908 Views
- 2 replies
- 0 kudos
Databricks Devops Release Pipeline Abort
We've built a release pipeline to our Databricks Workspaces, using the VNET Template. It's working end-to-end but intermittent aborts are occurring when workspace is recreated.For example, 4th of April (Monday) We recreated the workspaces and no abor...
- 7908 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Jonas Oliveira de Souza​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that be...
- 0 kudos
- 1168 Views
- 2 replies
- 0 kudos
Where is the dbc file for this couse :Â Deep Dive Into Lakehouse with Delta Lake
Where is the dbc file for this couse :Deep Dive Into Lakehouse with Delta Lake
- 1168 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @G S​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 0 kudos
- 4405 Views
- 1 replies
- 1 kudos
Database Objects Naming Convention for Bronze, Silver and Gold Layers
Please help me for database Objects Naming Convention and coding standard for Bronze, Silver and Gold Layers
- 4405 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Arun Sharma​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 1 kudos
- 6489 Views
- 5 replies
- 5 kudos
Resolved! Unity Catalog - Azure account console - how to access?
I'm trying to access account console in Azure but I only can see the list of workspaces and access them. I didn't find documentation about account console for Azure. Do you know how to access account console?
- 6489 Views
- 5 replies
- 5 kudos
- 5 kudos
Hello @Leszek​ . Please tell me is it works for you ?Did you find the root cause ?I still don't understand why I should grant to myself some extra permissions if I already global administrator, owner of subscription, owner of databricks workspace but...
- 5 kudos
- 2237 Views
- 2 replies
- 0 kudos
How to link 2 Dashboards
Hi, I have 2 dashboards. 1st Dashboard has a table with two columns, 'Country' and 'NumberOfMajorCities'. When I click on any row of this table/click on 'NumberOfMajorCities' field of a row, I would like to jump to 2nd Dashboard, where I display all ...
- 2237 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Ghazanfar Sl​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 3102 Views
- 3 replies
- 1 kudos
Error when creating a delta live table pipeline
INVALID_PARAMETER_VALUE: Validation failed for node_type_id, the value must be Standard_DS3_v2 (is "Standard_F8s")
- 3102 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Nitya Mehta​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 1 kudos
- 3237 Views
- 4 replies
- 0 kudos
Resolved! Datatype mismatch while reading data from sql server to databricks
Data from Azure sql server was read into databricks through JDBC connection (spark version 2.x) and stored into Gen1. Now the client wants to migrate the data from Gen1 to Gen2. When we ran the same jobs that read data from Azure Sql Server to Databr...
- 3237 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Mani Teja G​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...
- 0 kudos
- 3187 Views
- 6 replies
- 0 kudos
DLT autoloader credentials not available error in Azure
I'm just getting started with Databricks and DLTs. I've followed all the docs and tutorials I can find on this and believe I have set up everything in Azure correctly: service principal, paths, and spark configs. When I run a simple DLT autoloader pi...
- 3187 Views
- 6 replies
- 0 kudos
- 5163 Views
- 4 replies
- 1 kudos
Resolved! Ray dashboard no longer available
Has anyone else experienced the lack of access to the Ray dashboard since this week? Last week worked fine.rom ray.util.spark import setup_ray_cluster setup_ray_cluster(...)This used to output an HTML block with a link to the dashboard.I can manual...
- 5163 Views
- 4 replies
- 1 kudos
- 1 kudos
The reason for the missing dashboard for me was due ​to not having installed some required dependencies. Shout out to the Ray community for their help.​I've submitted a PR (now merged) to add a warning message​ when such dependencies are missing.​​
- 1 kudos
- 4762 Views
- 8 replies
- 0 kudos
When running DBT pipeline with column docs persisted we get error at least one column must be specified
Problem:When running dbt with persist column docs enabled we get the following error: org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the tableBackground:There is an issue on the dbt-spark github that was c...
- 4762 Views
- 8 replies
- 0 kudos
- 0 kudos
Also confirming that you do not have any of these limitations:From DBT's website: Some databases limit where and how descriptions can be added to database objects. Those database adapters might not support persist_docs, or might offer only partial su...
- 0 kudos
- 5879 Views
- 2 replies
- 0 kudos
`moduleNotFoundError` when attempting to enable a jupyter notebook extension
I am running a set of commands and to run `pydeck` on jupyter notebook as per the documentation here: https://pydeck.gl/installation.html#enabling-pydeck-for-jupyterHowever, it throws an `moduleNotFoundError` exception. !pip install pydeck !jupyter n...
- 5879 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Keval Shah​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 0 kudos