cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Hal
by New Contributor II
  • 1118 Views
  • 1 replies
  • 3 kudos

Connecting Power BI on Azure to Databricks on AWS?

Can someone share with me the proper way to connect Power BI running on Azure to Databricks running on AWS?

  • 1118 Views
  • 1 replies
  • 3 kudos
Latest Reply
bhanadi
New Contributor II
  • 3 kudos

Have the same question. Do we have to take care of any specific tasks to make it work. Anyone who implemented it?

  • 3 kudos
Saurabh707344
by New Contributor III
  • 17659 Views
  • 1 replies
  • 1 kudos

Platform and Approach Comparison

Do anyone have structure and crisp comparison between benefits of performing MLOps using below ways and what are the strong areas of each platform:a) Standalone Databricks where all pipelines and orchestration done on Databricks and external third pa...

  • 17659 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Saurabh Singh​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
pvignesh92
by Honored Contributor
  • 1445 Views
  • 1 replies
  • 3 kudos

lnkd.in

Databricks Auto Loader is an interesting feature that can be used to load data incrementally.✳ It can process new data files as they arrive in the cloud object stores✳ It can be used to ingest JSON, CSV, PARQUET, AVRO, ORC, TEXT and even Binary file ...

  • 1445 Views
  • 1 replies
  • 3 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 3 kudos

Thanks for sharing

  • 3 kudos
DianGermishuiz1
by New Contributor III
  • 2425 Views
  • 4 replies
  • 0 kudos

What is the purpose of the databricks-workspace-stack-lambdazipsbucket S3 bucket created on Databricks AWS Account Provisioning?

An S3 bucket with the prefix "databricks-workspace-stack-lambdazipsbucket" was created by default when I created my AWS Databricks account. It is set to public access. It has one zip file in it called "lambda.zip". What is the purpose of this S3 buck...

  • 2425 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Dian Germishuizen​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 0 kudos
3 More Replies
MShee
by New Contributor II
  • 1450 Views
  • 1 replies
  • 1 kudos
  • 1450 Views
  • 1 replies
  • 1 kudos
Latest Reply
NandiniN
Honored Contributor
  • 1 kudos

Hello @M Shee​ ,In a drop down you can select a value from a list of provided values, not type the values in. What you might be interested in is a combobox - It is combination of text and dropdown. It allows to select a value from a provided list or ...

  • 1 kudos
JordanYaker
by Contributor
  • 1857 Views
  • 1 replies
  • 1 kudos

What is the maximum number of workspaces per account using Databricks on AWS?

I've been looking through the documentation and I swear this used to be listed somewhere, but for the life of me I can't find it anymore.

  • 1857 Views
  • 1 replies
  • 1 kudos
Latest Reply
JordanYaker
Contributor
  • 1 kudos

Thanks @Kaniz Fatma​ 

  • 1 kudos
Ajay-Pandey
by Esteemed Contributor III
  • 1948 Views
  • 1 replies
  • 6 kudos

Variable explorer in Databricks With Databricks Runtime 12.1 and above, you can directly observe current Python variables in the notebook UI.To open t...

Variable explorer in DatabricksWith Databricks Runtime 12.1 and above, you can directly observe current Python variables in the notebook UI.To open the variable explorer, click in the right sidebar. The variable explorer opens, showing the value and ...

image Untitled
  • 1948 Views
  • 1 replies
  • 6 kudos
Latest Reply
jose_gonzalez
Moderator
  • 6 kudos

Thank you for sharing

  • 6 kudos
sonalitotade
by New Contributor II
  • 1630 Views
  • 2 replies
  • 0 kudos

Capture events such as Start, Stop and Terminate of cluster.

Hi,I am using databricks with AWS.I need to capture events such as Start, Stop and Terminate of cluster and perform some other action based on the events that happened on the cluster.Is there a way I can achieve this in databricks?

  • 1630 Views
  • 2 replies
  • 0 kudos
Latest Reply
sonalitotade
New Contributor II
  • 0 kudos

Hi Daniel, thanks for the responseI would like to know if we can capture the event logs as shown in the image below when an event occurs on the cluster.

  • 0 kudos
1 More Replies
rams
by Contributor
  • 2002 Views
  • 3 replies
  • 4 kudos

Resolved! 14 day trial version console showing blank screen after login

I have taken a trial version of Databricks and wanted to configure it with AWS. but after login it was showing as blank screen since 20 hours. can someone help me with this. Note: strictly i have to use AWS with Databricks for configuration.

  • 2002 Views
  • 3 replies
  • 4 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 4 kudos

try to reach your account manager

  • 4 kudos
2 More Replies
Manimkm08
by New Contributor III
  • 2339 Views
  • 3 replies
  • 0 kudos

Jobs are failed with AWS_INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET_FAILURE

We have assigned 3 dedicated subnets (one per AZ ) to the Databricks workspace each with /24 CIDR but noticed that all the jobs are running into a single subnet which causes AWS_INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET_FAILURE.Is there a way to segregat...

  • 2339 Views
  • 3 replies
  • 0 kudos
Latest Reply
Manimkm08
New Contributor III
  • 0 kudos

@karthik p​ Have configured one subnet per AZ(total 3). Have followed the same steps as mentioned in the document. Is there a way to check whether the Databricks uses all the subnets or not?@Debayan Mukherjee​ am not getting how to use LB in this set...

  • 0 kudos
2 More Replies
augustin
by New Contributor II
  • 4274 Views
  • 5 replies
  • 5 kudos

Mount an uncrypted AWS EFS in AWS Databricks

Hi,I want to mount an uncrypted AWS EFS in AWS Databricks. When I do:mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-abcdef.efs.region.amazonaws.com:/ /mnt/efs-uncryptedI get this error:mount.nfs4: moun...

  • 4274 Views
  • 5 replies
  • 5 kudos
Latest Reply
Andrei_Radulesc
Contributor III
  • 5 kudos

"To support NFS under LXC, some of the apparmor protections need to be lifted." (see https://theorangeone.net/posts/mount-nfs-inside-lxc/)

  • 5 kudos
4 More Replies
PaulP
by New Contributor II
  • 2184 Views
  • 3 replies
  • 6 kudos

What is the best expected starting time for a cluster when using a pool?

Hi! I'm doing some tests to get an idea of how much time could be saved starting a cluster by using a pool and was wondering if the results I get are what should be expected.We're using AWS Databricks and used i3.xlarge as instance type (if that matt...

  • 2184 Views
  • 3 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Paul Pelletier​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 6 kudos
2 More Replies
Manimkm08
by New Contributor III
  • 4194 Views
  • 4 replies
  • 4 kudos

Bootstrap Timeout during cluster start on AWS cloud

Sometimes am getting the below error when the cluster is started. Have attached the system log of the below mentioned instance from AWS. In recent days am getting this error for very frequently. Have seen same error is reported earlier and it marked ...

  • 4194 Views
  • 4 replies
  • 4 kudos
Latest Reply
Manimkm08
New Contributor III
  • 4 kudos

@Kaniz Fatma​ @karthik p​  Since morning, am facing the issue again. Seems the issue is intermittent and it fails the pipeline in mid of the ETL process. Couldn't able to get the exact root cause of the issue. Can someone provide what would be the wo...

  • 4 kudos
3 More Replies
swetha
by New Contributor III
  • 5069 Views
  • 2 replies
  • 2 kudos

Databricks job cluster logs

I am using databricks job cluster for multitask jobs, when my job failed/succeeded I couldn't see any logs, Do I need to add any location in advanced options, cluster logging to see the logs for the failed/succeeded jobs or what it is and how it work...

  • 5069 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @swetha kadiyala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 2 kudos
1 More Replies
fundat
by New Contributor II
  • 3122 Views
  • 3 replies
  • 1 kudos

Metastore - Access validation failed

Hi,When i create a metastore in aws databricks, i always have this error in the picture bellow.Eventhought i follow this link https://docs.databricks.com/data-governance/unity-catalog/get-started.html#cloud-tenant-setup-aws

metastore_databricks_error
  • 3122 Views
  • 3 replies
  • 1 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@Anatole Cadet​ looks your IAM role is not properly configured, can you please check

  • 1 kudos
2 More Replies
Labels