cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Ramana
by • Valued Contributor II
  • 4250 Views
  • 3 replies
  • 0 kudos

SHOW GROUPS is not giving groups available at the account level

I am trying to capture all the Databricks groups and their mapping to user/ad group(s).I tried to do this by using show groups, show users, and show grants by following the examples mentioned in the below article but the show groups command only fetc...

  • 4250 Views
  • 3 replies
  • 0 kudos
Latest Reply
Ramana
Valued Contributor II
  • 0 kudos

Yes, I can use the Rest API but I am looking for a SQL or Programming way to do this rather than doing the API calls and building the Comex Datatype Dataframe and then saving it as a Table.ThanksRamana

  • 0 kudos
2 More Replies
kseyser
by • New Contributor II
  • 3003 Views
  • 2 replies
  • 1 kudos

Predicting compute required to run Spark jobs

Im working on a project to predict compute (cores) required to run spark jobs. Has anyone work on this or something similar before? How did you get started? 

  • 3003 Views
  • 2 replies
  • 1 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 1 kudos

@kseyser good day, This documentation might help you in your use-case: https://docs.databricks.com/en/compute/cluster-config-best-practices.html#compute-sizing-considerations Kind regards, Yesh

  • 1 kudos
1 More Replies
Lea
by • New Contributor II
  • 7817 Views
  • 1 replies
  • 2 kudos

Resolved! Advice for generic file processing for ingestion of multiple data formats

Hello,We are using delta live tables to ingest data from multiple business groups, each with different input file formats and parsing requirements.  The input files are ingested from azure blob storage.  Right now, we are only servicing three busines...

  • 7817 Views
  • 1 replies
  • 2 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 2 kudos

Hello @Lea , I'd like to inform you that our platform does not currently provide a built-in feature for ingesting multiple or interchangeable file formats. However, we highly value your input and encourage you to share your ideas through Databricks' ...

  • 2 kudos
thiagoawstest
by • Contributor
  • 13834 Views
  • 2 replies
  • 1 kudos

Resolved! Migration Azure to AWS

Hello, today I use Azure Databricks, I want to migrate my wordspaces to AWS Databricks. What is the best practice, which path should I follow?, I didn't find anything in the documentation.thanks.

  • 13834 Views
  • 2 replies
  • 1 kudos
Latest Reply
thiagoawstest
Contributor
  • 1 kudos

Hello, as I already have a working Databricks environment on Azure, the best way would be to use tool-databricks-migrate?

  • 1 kudos
1 More Replies
deng_dev
by • New Contributor III
  • 4233 Views
  • 3 replies
  • 0 kudos

Autoloader ignore one folder in path

Hi everyone!I am trying to setup Autoloader to read json file with specific name from all subfolders under the path except one.Could someone advice how this can be achieved? For example, I need to read from .../*/specific_name.json, but ignore test f...

  • 4233 Views
  • 3 replies
  • 0 kudos
Latest Reply
standup1
Contributor
  • 0 kudos

I think you can use REGEXP to achieve this. This might not be the best way, but it should get the job done. It's all about filtering that file in the df from getting loaded. Try something like thisdf.select(“*”,”_metadata”).select(“*”,”_metadata.file...

  • 0 kudos
2 More Replies
Devsql
by • New Contributor III
  • 4711 Views
  • 3 replies
  • 2 kudos

How to find that given Parquet file got imported into Bronze Layer ?

Hi Team,Recently we had created new Databricks project/solution (based on Medallion architecture) having Bronze-Silver-Gold Layer based tables. So we have created Delta-Live-Table based pipeline for Bronze-Layer implementation. Source files are Parqu...

Data Engineering
Azure Databricks
Bronze Job
Delta Live Table
Delta Live Table Pipeline
  • 4711 Views
  • 3 replies
  • 2 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 2 kudos

Hello @Devsql , It appears that you are creating DLT bronze tables using a standard spark.read operation. This may explain why the DLT table doesn't include "new files" during a REFRESH operation. For incremental ingestion of bronze layer data into y...

  • 2 kudos
2 More Replies
youssefmrini
by • Databricks Employee
  • 6743 Views
  • 0 replies
  • 2 kudos

Delta Lake Liquid Clustering

Support for liquid clustering is now generally available using Databricks Runtime +15.2 Getting started with Delta Lake Liquid clustering https://lnkd.in/eaCZyhbF#DeltaLake #Databricks

  • 6743 Views
  • 0 replies
  • 2 kudos
pjv
by • New Contributor III
  • 5417 Views
  • 2 replies
  • 0 kudos

Asynchronous API calls from Databricks Workflow job

Hi all,I have many API calls to run on a python Databricks notebook which I then run regularly on a Databricks Workflow job. When I test the following code on an all purpose cluster locally i.e. not via a job, it runs perfectly fine. However, when I ...

  • 5417 Views
  • 2 replies
  • 0 kudos
Latest Reply
pjv
New Contributor III
  • 0 kudos

I actually got it too work though I do see that if I run two jobs of the same code in parallel the async execution time slows down. Do the number of workers of the cluster on which the parallel jobs are run effect the execution time of async calls of...

  • 0 kudos
1 More Replies
Olly
by • New Contributor II
  • 4265 Views
  • 2 replies
  • 1 kudos

Resolved! DBR14.3 Shared Access cluster delta.DeltaTable.toDF() issues

Having issues with the pyspark DataFrames returned by delta.DeltaTable.toDF(), in what I believe is specific to shared access clusters on DBR14.3. Recently created a near identical workflow with the only major difference being that one of the source ...

  • 4265 Views
  • 2 replies
  • 1 kudos
Latest Reply
Olly
New Contributor II
  • 1 kudos

That works, as mentioned it is easy to work around. as does replacing df = spark.table("test")df.select(df.col)

  • 1 kudos
1 More Replies
standup1
by • Contributor
  • 2966 Views
  • 2 replies
  • 0 kudos

How to exclude/skip a file temporarily in DLT

Hi,Is there any way to exclude a file from the dlt pipeline (autoload) run temporarily? What I mean is that I want to be able to exclude a specific file until I decided to include it in the load? I can't control the files or the location where they a...

  • 2966 Views
  • 2 replies
  • 0 kudos
Latest Reply
brockb
Databricks Employee
  • 0 kudos

Hi, I'm not aware of default Autoloader functionality that does what you're looking to do given that Autoloader is designed to incrementally ingest data as it arrives in cloud storage. Can you describe more about: "...exclude a specific file until I ...

  • 0 kudos
1 More Replies
VGS777
by • New Contributor III
  • 2322 Views
  • 1 replies
  • 0 kudos

Regarding Databricks Terraform to new user

Hey FolksI am new to terraform and databricksI have usecase I want to create new user or add them to databricks workspace. And assign role to this user. And also assign cluster to this new userAfter 12hrs I want to delete this new user and also these...

  • 2322 Views
  • 1 replies
  • 0 kudos
Latest Reply
VGS777
New Contributor III
  • 0 kudos

Thanks for this information 

  • 0 kudos
wyzer
by • Contributor III
  • 15595 Views
  • 8 replies
  • 4 kudos

Resolved! How to pass parameters in SSRS/Power BI (report builder) ?

Hello,In SSRS/Power BI (report builder), how to query a table in Databricks with parameters please ?Because this code doesn't works :SELECT * FROM TempBase.Customers WHERE Name = {{ @P_Name }}Thanks.

  • 15595 Views
  • 8 replies
  • 4 kudos
Latest Reply
Nj11
New Contributor II
  • 4 kudos

Hi, I am not able to see the data in SSRS while I am using date parameters but with manual dates data is populating fine. The database is pointing to databricks. I am not sure what I am missing here. Please help me in this. ThanksI am trying with que...

  • 4 kudos
7 More Replies
mh_db
by • Databricks Partner
  • 3517 Views
  • 2 replies
  • 0 kudos

Unable to connect to oracle server from databricks notebook in AWS

I'm trying to connect to oracle server hosted in azure from AWS databricks notebook but seems the connection keeps timing out. I tested the connection IP using telnet <hostIP> 1521 command from another EC2 instance and that seems to reach the oracle ...

Data Engineering
AWS
oracle
TCP
  • 3517 Views
  • 2 replies
  • 0 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 0 kudos

@mh_db good day! Could you please confirm the Cluster type you used for testing? Was it a Shared Cluster, an Assigned/Single-User Cluster, or a No-Isolation cluster? Could you please try the same on the Assigned/Single User Cluster and No Isolation c...

  • 0 kudos
1 More Replies
Labels