- 2367 Views
- 1 replies
- 0 kudos
Row Level Validation
I have two array one of devl other one is prod.Inside this there are many tables .How do i compare and check the count difference.Wanted to create a automated script so as to check the count difference and perform row level validation.Pyspark script...
- 2367 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, You can use except command for the same. Please refer: https://stackoverflow.com/questions/70366209/databricks-comparing-two-tables-to-see-which-records-are-missing. Please let us know if this helps.
- 0 kudos
- 2286 Views
- 2 replies
- 1 kudos
Databricks SQL permission problems
We are using a catalog and normally I have the ALL PREVELAGES user status but Im not able to modify the SQL script which is created by some of my colleagues. They have to give me an access and after that Im able to modify. How can I solve this proble...
- 2286 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, when you are not able to modify could you please confirm the error you are receiving? Also, you can refer to https://docs.databricks.com/_static/notebooks/set-owners-notebook.html and https://docs.databricks.com/sql/admin/transfer-ownership.html
- 1 kudos
- 9684 Views
- 5 replies
- 6 kudos
API to write into Databricks tables
Hello Experts,Is there any API in databricks that allows to write the data in the Databricks tables. I would like to send small size Logs information to Databricks tables from other service. What are my options?Thank you very much.
- 9684 Views
- 5 replies
- 6 kudos
- 6 kudos
and what about use the jdbc/odbc driver, either programatically or using a tool like dbeaver?​
- 6 kudos
- 6258 Views
- 2 replies
- 4 kudos
Resolved! GitCopilot Support
Any Idea why GitCopiot is not available in Azure Databricks, though it supports Github?
- 6258 Views
- 2 replies
- 4 kudos
- 4 kudos
That is true (this is not an answer but a comment) vscode is supported. But vscode is not integrating with notebook on aws. When will this feature be available?
- 4 kudos
- 3928 Views
- 3 replies
- 0 kudos
Resolved! For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected?
If Yes, how is order ensured? For example, let's say there are a number of CDC change files that are uploaded to a directory over time. If a table were to be created using the cloudFiles source, in what order would those files be processed?
- 3928 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, Is there any new development in terms of ensuring ordering of the files in autoloader?
- 0 kudos
- 19716 Views
- 5 replies
- 17 kudos
Resolved! python wheel cannot be installed as library.
When I try to install the python whl library, I get the below error. However I can install it as a jar and it works fine. One difference is that I am creating my own cluster by cloning an existing cluster and copying the whl to a folder called testin...
- 19716 Views
- 5 replies
- 17 kudos
- 17 kudos
The issue was that the package was renamed after it was installed to the cluster and hence it was not recognized.
- 17 kudos
- 1327 Views
- 0 replies
- 0 kudos
DLT using the result of one view in another table with collect()
Hey,Do you guys know, if there is an option to implement something like this in DLT:@dlt.view()def view_1(): # some calculations that return a small dataframe with around max 80 rows@dlt.table()def table_1(): result_df = dlt.read("view_1") resu...
- 1327 Views
- 0 replies
- 0 kudos
- 4183 Views
- 4 replies
- 2 kudos
Accreditation, Badges, Points not received
Hi there​ ,I have completed a few courses but didn't receive any badges or points. I also did an accreditation but also didn't receive anything. It's been already 3 or 4 days and still nothing.I would really appreciate if Databricks could fix this.Ma...
- 4183 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Andre Paiva​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 2 kudos
- 10494 Views
- 3 replies
- 8 kudos
Resolved! Advantages of Databricks Lakehouse over Azure synapse.
What are more advantages of data bricks over azure synapse analytics. Looks like most of them are almost similar features like computation or storage etc... in both.
- 10494 Views
- 3 replies
- 8 kudos
- 8 kudos
Below link has good comparison of both:https://hevodata.com/learn/azure-synapse-vs-databricks/
- 8 kudos
- 2561 Views
- 2 replies
- 0 kudos
Spark pushdown filter not being respected on dbfs
I have a parquet files with a column g1 with schemaStructField(g1,IntegerType,true)Now I have a query with filter on g1.What's weird in the SQL viewer is that spark is loading all the rows from that file. Even though in the physical plan I can see th...
- 2561 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks @Ajay Pandey​ pls find attached the physical plan.Query: Select identityMap, segmentMembership, _repo, workEmail, person, homePhone, workPhone, workAddress, personalEmail, homeAddress from final_segment_index_table_v2 where (g1 >= 128 AND g1 <...
- 0 kudos
- 8516 Views
- 3 replies
- 3 kudos
Resolved! Data Engineering - CTAS - External Tables - Limitations of CTAS for external tables - can or cannot use options and location
Data Engineering - CTAS - External TablesCan someone help me understand why In chapter 3.3, we cannot not directly use CTAS with OPTIONS and LOCATION to specify delimiter and location of CSV?Or I misunderstood?Details:In Data Engineering with Databri...
- 8516 Views
- 3 replies
- 3 kudos
- 3 kudos
The 2nd statement CTAS will not be able to parse the csv in any manner because it's just the from statement that points to a file. It's more of a traditional SQL statement with select and from. It will create a Delta Table. This just happens to b...
- 3 kudos
- 6262 Views
- 2 replies
- 2 kudos
SQL table alias autocomplete
I have a table with 600 columns and the table name is long. I want to use a table alias with autocomplete but it's not working. Any ideas how I can get this to work? works%sql --autocomplete works SELECT verylongtablename.column200 verylongtabl...
- 6262 Views
- 2 replies
- 2 kudos
- 2 kudos
My cluster is running fine. Does autocomplete work for you with a table alias?
- 2 kudos
- 4828 Views
- 2 replies
- 1 kudos
Table of Content consistency
When I click on header "STEP 3" in the table of contents, it takes me to the correct section. However, when I click on "STEP 2" - the table of contents stays on "STEP 3". This sometime causes confusion. For consistency, is there any way to highligh...
- 4828 Views
- 2 replies
- 1 kudos
- 1 kudos
If you click on cell "Command-4", does the table of contact (on the left) highlight "Command-4"?
- 1 kudos
- 3068 Views
- 1 replies
- 2 kudos
- 3068 Views
- 1 replies
- 2 kudos
- 2 kudos
because your driver is not able to talk with your nodes for this you can add configuration where you can increase databricks heartbeat interval and you can also add rpc max size this will also help.you can explore cluster configuration from here- htt...
- 2 kudos
- 1896 Views
- 1 replies
- 0 kudos
A Job "pool"? (or task pool)
I'm trying to run a single job multiple times with different parameters where the number of concurrent jobs is less than the number of parameters.I have a job (or task...) J that takes parameter set p, I have 100 p values I want to run, however I onl...
- 1896 Views
- 1 replies
- 0 kudos
- 0 kudos
this is something new,interesting question, try to reach out databricks support team, maybe they have some good idea here
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
