- 31773 Views
- 4 replies
- 4 kudos
How to extract DDL from tables/views using a SQL query?
I want to write a SQL query that queries the information_schema to generate a list of objects, their columns, relationships etc. - basically a data dictionary. For each object I want to show the DDL code, and I know I can get it by executing show c...
- 31773 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Richard Architect​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I sugg...
- 4 kudos
- 3273 Views
- 2 replies
- 2 kudos
Resolved! Delta sharing (databricks-to-databricks) between azure regions issue
we have 2 unity meta stores in 2 regions ( useast2 contains data and westeurope contains clusters) and enabled delta sharing between them. We use azure storage firewall / vent whitelisting to allow secure connection to storage from compute cluste...
- 3273 Views
- 2 replies
- 2 kudos
- 2 kudos
@prasad vaze​ :Delta sharing between Unity metastores in different regions can be achieved in several ways, depending on your specific requirements and constraints. One common approach is to use Azure Private Link to establish a private connection be...
- 2 kudos
- 5477 Views
- 3 replies
- 1 kudos
dbsqlcli.main ERROR - getaddrinfo failed
I had to rebuild my laptop and am trying to re-install Databricks and Databricks-sql-cli so I can use the CLI to pull info from the databricks repository.When I try to run the command: dbsqlcli -e "select distinct tableName from databases_info.all_db...
- 5477 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @John Zajic​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1 kudos
- 3514 Views
- 2 replies
- 1 kudos
Resolved! Pyspark read multiple Parquet type expansion failure
ProblemReading nearly equivalent parquet tables in a directory with some having column X with type float and some with type double fails.Attempts at resolvingUsing streaming filesRemoving delta caching, vectorizationUsing ,cache() explicitlyNotesThis...
- 3514 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Erik Louie​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Regards
- 1 kudos
- 4121 Views
- 6 replies
- 4 kudos
Databricks Attribute Error: 'IPythonShell' object has no attribute 'kernel'
I have been getting this error repeatedly when trying to run a notebook. I have tried attaching multiple different clusters and installing some of the libraries that it wanted me to update. I have also tried to clear the state of the notebook and res...
- 4121 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi @Toshali Mohapatra​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 4 kudos
- 3515 Views
- 2 replies
- 8 kudos
Databricks has announced that users can now create notebooks in Jupyter format in Repos, offering a familiar experience for creating and editing noteb...
Databricks has announced that users can now create notebooks in Jupyter format in Repos, offering a familiar experience for creating and editing notebooks. This update allows users to integrate with the broader data science ecosystem, import and expo...
- 3515 Views
- 2 replies
- 8 kudos
- 8 kudos
Hi @Hubert Dudek​ Thanking you for helping us to build a vibrant and resourceful community by recognizing and highlighting insightful contributions.Regards
- 8 kudos
- 1108 Views
- 2 replies
- 0 kudos
- 1108 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @令辉 å”​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback will he...
- 0 kudos
- 1890 Views
- 2 replies
- 1 kudos
Cosmos DB Connector for 12.1 Cluster and above
Hi guys,​You know what`s the version suporte the databricks cluster version above 12.1 ?my cluster:error: Thank you
- 1890 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @William Scardua​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 1 kudos
- 2014 Views
- 1 replies
- 0 kudos
Resolved! TypeError on DataFrame via spark readStream transform invocation of UDF
Our use case is to "clean up" column names (remove spaces, etc) on ingestion of CSV data using the Delta Live Table capability. We desire to use the schema inference capability during ingestion so schema specification (up front) will not be happenin...
- 2014 Views
- 1 replies
- 0 kudos
- 0 kudos
The issue was erroneously believing the transform function needed UDF decorator. With the decorator removed the transform invokes (and works) as expected.
- 0 kudos
- 1785 Views
- 3 replies
- 0 kudos
Performance issue while loading bulk data into Post Gress DB from data bricks.
We are facing a performance issue while loading bulk data into Postgress DB from data bricks. We are using spark JDBC connections to move the data. However, the rate of transfer is very low which is causing performance bottleneck. is there any better...
- 1785 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @Janga Reddy​ @Daniel Sahal​ and @Vidula Khanna​ ,To enhance performance in general we need to design for more parallelism, in Spark JDBC context this controlled by the number of partitions for the data to be writtenThe example here shows how t...
- 0 kudos
- 653 Views
- 0 replies
- 1 kudos
������#ASEANlakehouse
#ASEANlakehouse
- 653 Views
- 0 replies
- 1 kudos
- 3057 Views
- 2 replies
- 2 kudos
Resolved! I am new to data bricks. setting up Workspace for NON-prod environment Separate workspaces for DEV, QA or Just one work space for NON-prod ?
What i learned based on learning materials, documents, etc.. For data bricks it is a good practice to set up 1 non-prod workspace but separate clusters for Dev, QA, SIT, etc.Is it best practice to set up only 1 NON-PROD Workspace instead of separate ...
- 3057 Views
- 2 replies
- 2 kudos
- 3487 Views
- 4 replies
- 2 kudos
SAT - Security Analysis Tool implementation error
I want to implement SAT in my workspace account. I was able to execute the terraform that enable the necessary infra to work on that. When I try to execute the workflow "SAT Initializer Notebook (one-time)" it fails with the error:AnalysisException: ...
- 3487 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Arnold Souza​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 2 kudos
- 2719 Views
- 1 replies
- 7 kudos
SQL cells in databricks notebooks can now be run in parallel, which means faster query processing and analysis. This new feature is especially helpful...
SQL cells in databricks notebooks can now be run in parallel, which means faster query processing and analysis. This new feature is especially helpful for queries that take longer to run or analyze large datasets. With parallel processing, Databricks...
- 2719 Views
- 1 replies
- 7 kudos
- 7 kudos
Informative ​
- 7 kudos
- 11649 Views
- 1 replies
- 1 kudos
Resolved! MERGE to update a column of a table using Spark SQL
Coming from MS SQL background, I'm trying to write a query in Spark SQL that simply update a column value of table A (source table) by INNER JOINing a new table B with a filter.MS SQL query looks like this:UPDATE T SET T.OfferAmount = OSE.EndpointEve...
- 11649 Views
- 1 replies
- 1 kudos
- 1 kudos
Posting answer to my question: MERGE into TempOffer VIEW USING OfferSeq OSE ON VIEW.OfferId = OSE.OfferID AND OSE.OfferId = 1 WHEN MATCHED THEN UPDATE set VIEW.OfferAmount = OSE.EndpointEventAmountValue;
- 1 kudos