cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

DBX-Beginer
by New Contributor
  • 3321 Views
  • 2 replies
  • 0 kudos

Display count of records in all tables in hive meta store based on one of the column value.

I have a DB name called Test in Hive meta store of data bricks. This DB contains around 100 tables. Each table has the column name called sourcesystem and many other columns. Now I need to display the count of records in each table group by source sy...

  • 3321 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krish K​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
doalmeida
by New Contributor
  • 2422 Views
  • 2 replies
  • 0 kudos

Endpoint not found when creating automl experiment through API and UI

Hi everyone, I'm getting this exact error as shown bellow, when trying to create an automl experiment. This happens both through the UI and the API with my code or with databrick's example code. I've tried looking into this but had no luck finding an...

image
  • 2422 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Diogo Almeida​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 0 kudos
1 More Replies
Dataengineer_mm
by New Contributor
  • 3724 Views
  • 2 replies
  • 0 kudos

Passing a date parameter through workflow

Hi , when we pass the parameter through workflows in DB, should we need to manually provide the parameter all the time? or any dynamic way of passing the parameter?

  • 3724 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Menaka Murugesan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 0 kudos
1 More Replies
Pankaj2009
by New Contributor II
  • 1533 Views
  • 5 replies
  • 0 kudos

FDBAv2 certificate

Hi Team I haven't received my data bricks vouchers as i have already submitted the Fundamentals of the databricks lakehouse platform V2 . Name: Pankaj Kumar Goyalemail: pankajaggarwal98@gmail.com

  • 1533 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Pankaj kumar Goyal​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...

  • 0 kudos
4 More Replies
KiavData_78669
by New Contributor II
  • 829 Views
  • 3 replies
  • 0 kudos

Is there a native way to have multiple versions of the same notebook/cluster/workflow?

Notebooks code, clusters and workflows definitions can evolve through time. When used in production, I would like to support old versions. Is there a way to run a job referring to a certain notebook version, using a given cluster version? Of course, ...

  • 829 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Kiavi Data​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 0 kudos
2 More Replies
Pazuzu7
by New Contributor II
  • 941 Views
  • 3 replies
  • 0 kudos

Upgrading to 11.3lts, Sedona functions throwing null when previously worked fine in 7.3

I'm in the process of upgrading to 11.3. I'm using spark 3.3.0, scala 2.12, maven and sedona 1.2.0 incubating and followed the installation as outlined by sedona here. Everything was running smoothly in version 7.3 but is currently throwing when reac...

  • 941 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @William Honeyman​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...

  • 0 kudos
2 More Replies
iwan_aucamp
by New Contributor III
  • 1147 Views
  • 2 replies
  • 1 kudos

Account SCIM API OpenAPI specification issues

I'm trying to get a list of all users, groups and service principals on Azure from a python script. As I understand things I should be using the Account SCIM API for this. According to the azure documentation [ref], the OpenAPI specification for this...

  • 1147 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Iwan Aucamp​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 1 kudos
1 More Replies
Rahul2025
by New Contributor III
  • 3618 Views
  • 11 replies
  • 1 kudos

Limitation on size of init script

Hi,We're using Databricks Runtime version 11.3LTS and executing a Spark Java Job using a Job Cluster. To automate the execution of this job, we need to define (source in from bash config files) some environment variables through an init script (clust...

  • 3618 Views
  • 11 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Rahul K​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your ...

  • 1 kudos
10 More Replies
Anonymous
by Not applicable
  • 1482 Views
  • 4 replies
  • 0 kudos

Objective is to make table unique at ID using group by , concat_ws and collect_list ,combining distinct values in one row.

Objective is to make table unique at ID. Table structure is as in attached image.Query used is : selectID,concat_ws(' & ' , collect_list(Distinct Gender)) as Genderfrom tablegroup by IDIt can be possible if we can order values within collect_list and...

  • 1482 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Rishabh Shanker​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 0 kudos
3 More Replies
Jerry01
by New Contributor III
  • 720 Views
  • 2 replies
  • 0 kudos

Is writing custom function possible in transform(array,func) in databricks sql?

This is the query I am trying to implementCreate function data_hide(data string)Return if(is_member('groupName'),data,'****')​Table : my_tableId Subject​1. ['Eng','Bio']2. ['Phy','Mat']​Select id, transform(Subject, x -> data_hide(x)) as new_data...

  • 720 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Naveena G​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...

  • 0 kudos
1 More Replies
mrcity
by New Contributor II
  • 1514 Views
  • 2 replies
  • 1 kudos

Exclude absent lookup keys from dataframes made by create_training_set()

I've got data stored in feature tables, plus in a data lake. The feature tables are expected to lag the data lake by at least a little bit. I want to filter data coming out of the feature store by querying the data lake for lookup keys out of my inde...

  • 1514 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Stephen Wylie​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 1 kudos
1 More Replies
goal1860
by New Contributor III
  • 2619 Views
  • 5 replies
  • 2 kudos

Resolved! Failed to signup community version

I've been trying to create Community Edition account, but keep getting: "An error has occurred. Please try again later" message. I searched the other posts, there are some people running into the same issue as well, but don't see any solution posted....

Screen Shot 2023-02-03 at 9.13.33 AM
  • 2619 Views
  • 5 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Liang He​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 2 kudos
4 More Replies
tibfab
by New Contributor II
  • 3262 Views
  • 5 replies
  • 0 kudos

How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)?

I successfully built a custom docker image for the Standard runtime following the steps described on the page Customize containers with Databricks Container Services and based on the image databricksruntime/standard:11.3-LTS. However, I cannot find ...

  • 3262 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Tibor Fabian​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!

  • 0 kudos
4 More Replies
nolanlavender00
by New Contributor
  • 2746 Views
  • 2 replies
  • 0 kudos

How to control garbage collection while using Autoloader File Notification?

I am using Autoloader to load files from a directory. I have set up File Notification with the Event Subscription. I have a backfill interval set to 1 day and have not run the stream for a week. There should only be about ~100 new files to pick up an...

  • 2746 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @nolanlavender008​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...

  • 0 kudos
1 More Replies
joshi
by New Contributor II
  • 1881 Views
  • 5 replies
  • 0 kudos

Full screen video' button not working in spark certification videos

Hi All,Many users already posted about this but no action taken till now., i tried to use different browsers and system still not able to maximize the spark training videos.Many months passed still databricks people are not correcting this mistake. @...

databricks
  • 1881 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Abhishek Joshi​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!

  • 0 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels