- 14091 Views
- 4 replies
- 2 kudos
Rename the file in Databricks is so hard.How to make it simpler
Hi Community Actually my requirement is simple , I need to drop the files into Azure data Lake gen 2 storage from Databricks. But When I use df.coalesce(1).write.csv("url to gen 2/stage/) It's creating part .CSV file . But I need to rename to a cust...
- 14091 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Philospher1425, Allow me to clarify that dbutils.fs serves as an interface to submit commands to your cloud provider storage. As such, the speed of copy operations is determined by the cloud provider and is beyond Databricks' control. That be...
- 2 kudos
- 6245 Views
- 6 replies
- 1 kudos
Tables disappear when I re-start a new cluster on Community Edition
What would be the best plan for independent course creator?Hi folks! I want to use databrick community edition as the platform to teach online courses. As you may know, for community edition, you need to create a new cluster when the old one terminat...
- 6245 Views
- 6 replies
- 1 kudos
- 1 kudos
I believe only the metadata get's removed from HMS not the delta files from dbfs. Instead of loading the data again and again try using ctas with that dbfs location.
- 1 kudos
- 1815 Views
- 0 replies
- 0 kudos
Information_schema appears empty
Hi,We've encountered a problem with the information schema for one of our catalogs. For context: we're using dbt to implement our logic. We noticed this issue because dbt queries the information_schema.tables view to check which tables should be drop...
- 1815 Views
- 0 replies
- 0 kudos
- 2580 Views
- 3 replies
- 3 kudos
- 2580 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @venkateshgunda, Community Edition managed storage is temporary.
- 3 kudos
- 2694 Views
- 2 replies
- 0 kudos
which cluster/worker/driver type is best for analytics work?
which cluster/worker/driver type is best for analytics work?
- 2694 Views
- 2 replies
- 0 kudos
- 0 kudos
Analytics work as in querying and analyzing data? Preferably using Databricks SQL? If so, then a SQL Warehouse is your best friend.
- 0 kudos
- 5519 Views
- 5 replies
- 2 kudos
Resolved! Delta sharing open protocol in Unity catalog: FileNotFoundError
Hi TeamI have created a recipient under delta sharing (azure databricks) . Unity catalog is enabled and data is stored in ADLS gen2. I have downloaded the credential file and trying to resue in my python script (as per databricks documentation) for a...
- 5519 Views
- 5 replies
- 2 kudos
- 2 kudos
I wasn't able to reproduce your issue. Is your delta table operable? can you see sample data from within databricks and query the table from within databricks? It almost looks like some parquet files are missing, causing your delta not queryable anym...
- 2 kudos
- 1692 Views
- 1 replies
- 0 kudos
Issues with Common Data Model as Source - different column size for blobs
I have a Dataverse Synapse link set up to extract data into ADLS gen2. I am trying to connect ADLS gen2 as the data source to read the data files in Databricks. I have CDC enabled for CDM Data with the partition of Year and Month.So, for example, if ...
- 1692 Views
- 1 replies
- 0 kudos
- 0 kudos
This is a thoughtful consideration, but have you considered using .option("mergeSchema", "true") when writing?Do keep in mind that this will affect the target table and possible downstream consumers. Ideally you want to have strict a schema contract ...
- 0 kudos
- 6362 Views
- 2 replies
- 0 kudos
Resolved! How can we connect to 2 different hive spark.hadoop.hive.metastore.uris
We need to read a table from 2 different spark.hadoop.hive.metastore.uris and do some validations.We are not able to connect to both spark.hadoop.hive.metastore.uris at the same time using sparkSession.I will be using Spark version: 3.1.1 and the lan...
- 6362 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi there @maskepravin02,We have once implemented this approach of two reading two different hive metasores, but it was not on AWS and GCP, maybe the docs can help.Though it is not recommended The best approach is to create separate spark applications...
- 0 kudos
- 1509 Views
- 1 replies
- 0 kudos
how do i pass the value from one task in the workflow to another task?
how do i pass the value from one task in the workflow to another task? #worksflow #orchestration
- 1509 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pt07 , This may be what you're looking for. Can you please take a look? https://www.databricks.com/blog/2022/08/02/sharing-context-between-tasks-in-databricks-workflows.html
- 0 kudos
- 2962 Views
- 1 replies
- 0 kudos
- 2962 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta Uniform is the way to go at the moment: https://www.databricks.com/blog/delta-lake-universal-format-uniform-iceberg-compatibility-now-ga In one word, tables are written in Delta but have compatibility with Iceberg as they will also save iceberg...
- 0 kudos
- 737 Views
- 0 replies
- 0 kudos
Interesting talks at DAIS 2024
I feel some of the talks are quite interesting and inspiring. Great to see everyone here!
- 737 Views
- 0 replies
- 0 kudos
- 1651 Views
- 2 replies
- 1 kudos
Liquid Clustering
Love the new concept, will save a lot of compute and make queries a lot faster!
- 1651 Views
- 2 replies
- 1 kudos
- 593 Views
- 0 replies
- 0 kudos
Training sessions are helpful
Streaming and Unity catalog sessions are awesome
- 593 Views
- 0 replies
- 0 kudos
- 750 Views
- 0 replies
- 0 kudos
Data and AI Summit
The Data and AI Summit is San Francisco is wonderful!
- 750 Views
- 0 replies
- 0 kudos
- 1517 Views
- 1 replies
- 0 kudos
DataAIsummit 2024
Having a wonderful learning experience and sharing experience and knowledge with fellow data engineers/scientists/architects
- 1517 Views
- 1 replies
- 0 kudos
- 0 kudos
Interesting use cases so far. Looking forward ot the sessions tomorrow!
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
