- 8187 Views
- 3 replies
- 1 kudos
Best way to generate fake data using underlying schema
HiWe are trying to generate fake data to run our tests. For example, we have a pipeline that creates a gold layer fact table form 6 underlying source tables in our silver layer. We want to generate the data in a way that recognises the relationships ...
- 8187 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Nick_Hughes This may be late for your scenario - but hopefully others facing similar issues will find it useful.You can specify how data is generated in `dbldatagen` using rules in the data generation spec. If rules are specified for data generat...
- 1 kudos
- 35013 Views
- 7 replies
- 13 kudos
Resolved! How to read data from a table into a dataframe outside of Databricks environment?
Hi, I am working on an ML project and I need to access the data in tables hosted in my Databricks cluster through a notebook that I am running locally. This has been very easy while I run the notebooks in Databricks but I cannot figure out how to do ...
- 35013 Views
- 7 replies
- 13 kudos
- 13 kudos
We can use Apis and pyodbc to achieve this. Once go through the official documentation of databricks that might be helpful to access outside of the databricks environment.
- 13 kudos
- 1204 Views
- 1 replies
- 0 kudos
External Tables - SQL
To create external tables we need to use the location keyword and use the link for the storage location, in reference to that does the user need to have permission for the storage location if not then will we use storage credentials to provide the ac...
- 1204 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Nimai, That's partially right. You can grant permissions directly on the storage credential, but Databricks recommends that you reference it in an external location and grant permissions to that instead. An external location combines a storage cre...
- 0 kudos
- 4149 Views
- 2 replies
- 0 kudos
Display count of records in all tables in hive meta store based on one of the column value.
I have a DB name called Test in Hive meta store of data bricks. This DB contains around 100 tables. Each table has the column name called sourcesystem and many other columns. Now I need to display the count of records in each table group by source sy...
- 4149 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Krish K​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 1557 Views
- 3 replies
- 0 kudos
Not able to perform update in delta table in databricks using 3 tables
Hi,I am able to perform merge from 2 tables but have requirement to update table based on 3 tables like following query.update a set a.name=b.namefrom table1 a inner join table2 b on a.id=b.idinner join table3 c on a.id=c.idThanks in advance..
- 1557 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @upendra kumar sharma​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Thanks and Regards
- 0 kudos
- 3219 Views
- 2 replies
- 0 kudos
Can't access delta tables under SQL Warehouse cluster. Getting Error while using path .../_delta_log/000000000.checkpoint
In our Databricks workspace, we have several delta tables available in the hive_metastore catalog. we are able to access and query the data via Data Science & Engineering persona clusters with no issues. The cluster have the credential passthrough en...
- 3219 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Rafael Gomez​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
- 0 kudos
- 4098 Views
- 2 replies
- 2 kudos
Issue with reading exported tables stored in parquet
Hi All, I am exported all tables from postgres snapshot into S3 in parquet format. I am trying to read the table using databricks and i am unable to do so. I get the following error: "Unable to infer schema for Parquet. It must be specified manually....
- 4098 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @shiva charan velichala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that bes...
- 2 kudos
- 2947 Views
- 5 replies
- 0 kudos
Delta Live Tables data quality rules application.
I have a requirement, where I need to apply inverse DQ rule on a table to track the invalid data. For which I can use the following approach:import dltrules = {}quarantine_rules = {}rules["valid_website"] = "(Website IS NOT NULL)"rules["valid_locatio...
- 2947 Views
- 5 replies
- 0 kudos
- 0 kudos
You can get additional info from DLT event log which is in delta so you can load it as table https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-event-log.html#data-quality
- 0 kudos
- 2582 Views
- 2 replies
- 2 kudos
How to check/list the tables which has CDF enabled?
How to list all the tables which has CDF enabled?I can review a table to find out if CDF is enable with below code.SHOW TBLPROPERTIES tableA(delta.enableChangeDataFeed)The return key ...
- 2582 Views
- 2 replies
- 2 kudos
- 2 kudos
Hhi @Tim zhang​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 2 kudos
- 2459 Views
- 2 replies
- 2 kudos
How to delete duplicate tables?
Hi Everyone,Accidently imported duplicate tables, guide me how to delete themusing data bricks community edition
- 2459 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Mahesh Babu Uppala​ You can use the following method to delete only the duplicate tables%scala val tables = spark.sql("""SHOW TABLES""").createOrReplaceTempView("tables") val temp_tables = spark.sql("""select tableName from tables where tableName...
- 2 kudos
- 17059 Views
- 5 replies
- 7 kudos
Resolved! Best Practices for naming Tables and Databases in Databricks
We moved in Databricks since few months from now, and before that we were in SQL Server.So, all our tables and databases follow the "camel case" rule.Apparently, in Databricks the rule is "lower case with underscore".Where can we find an official doc...
- 17059 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi @Salah KHALFALLAH​ , looking at the documentation it appears that Databricks' preferred naming convention is lowercase and underscores as you mentioned.The reason for this is most likely because Databricks uses Hive Metastore, which is case insens...
- 7 kudos
- 4547 Views
- 5 replies
- 6 kudos
API to write into Databricks tables
Hello Experts,Is there any API in databricks that allows to write the data in the Databricks tables. I would like to send small size Logs information to Databricks tables from other service. What are my options?Thank you very much.
- 4547 Views
- 5 replies
- 6 kudos
- 6 kudos
and what about use the jdbc/odbc driver, either programatically or using a tool like dbeaver?​
- 6 kudos
- 5310 Views
- 3 replies
- 3 kudos
Resolved! Data Engineering - CTAS - External Tables - Limitations of CTAS for external tables - can or cannot use options and location
Data Engineering - CTAS - External TablesCan someone help me understand why In chapter 3.3, we cannot not directly use CTAS with OPTIONS and LOCATION to specify delimiter and location of CSV?Or I misunderstood?Details:In Data Engineering with Databri...
- 5310 Views
- 3 replies
- 3 kudos
- 3 kudos
The 2nd statement CTAS will not be able to parse the csv in any manner because it's just the from statement that points to a file. It's more of a traditional SQL statement with select and from. It will create a Delta Table. This just happens to b...
- 3 kudos
- 4771 Views
- 4 replies
- 8 kudos
Is it possible to create a sqlite database and export it?
I am trying to create a sqlite database in databricks and add a few tables to it. Ultimately, I want to export this using Azure. Is this possible?
- 4771 Views
- 4 replies
- 8 kudos
- 8 kudos
@Hubert Dudek​ We currently have a process in place that reads in a SQLite file. We recently transitioned to using Databricks. We were hoping to be able to create a SQLite file so we didn't have to alter the current process we have in place.
- 8 kudos
- 4451 Views
- 3 replies
- 2 kudos
Could jobs do everything delta live tables do ?
Hello,I've read the posts:Jobs - Delta Live tables difference (databricks.com)andDifference between Delta Live Tables and Multitask Jobs (databricks.com)My understanding is that delta live tables are more like a DSL that simplfies the workflow defini...
- 4451 Views
- 3 replies
- 2 kudos
- 2 kudos
@Landan George​ "Jobs won't be able to do what DLT does", I read some blogs, and watched some videos too, but I still cannot figure out the difference between jobs vs DLT. Does it mean without Databricks DLT, Databricks jobs cannot handle delta table...
- 2 kudos